CNTRLPLANE-3509: Resolve N-minor release images via release controller API (v2) by jparrill · Pull Request #80360 · openshift/release

jparrill · 2026-06-10T14:19:28Z

Summary

Replace ci-operator release imports (n1minor through n4minor) with runtime resolution via the release controller REST API
Fixes the permafail introduced by CNTRLPLANE-3509: Resolve N-minor release images via release controller API #79627 (reverted in Revert "hypershift: resolve nodepool releases at runtime" (#79627) #80349) where unguarded variable expansion under set -o nounset crashed run-e2e scripts before tests could execute
Removes unused latest-418/419/420 release definitions that were left orphaned after removing NX dependencies from 4.22 jobs
Removes unused NODEPOOL_NX_VERSION env vars from e2e-aws-autonode (Karpenter-only job, does not run N-minor tests)

Context

PR	Status	Description
#79627	Merged (Jun 5)	Original implementation — caused permafail due to nounset-unsafe `${OCP_IMAGE_NX}` expansion
#80349	Merged (Jun 10)	Revert of #79627 to restore CI stability
This PR	—	v2 with fixes for the root cause + nightly fallback

What changed vs #79627

Nounset-safe variable expansion: All run-e2e scripts now use ${OCP_IMAGE_NX:-} with -n check instead of bare ${OCP_IMAGE_NX} — prevents crash when resolve step fails or variable is unset
Image availability verification: The resolve step verifies each resolved image is actually pullable from the registry using oc image info before accepting it. This catches stale/garbage-collected CI images.
Nightly stream fallback: When a CI stream image has been garbage-collected (e.g. 4.19 CI has no Accepted build in 8+ days), the resolve step automatically falls back to the nightly stream's latest Accepted release.
Removed orphaned release definitions: latest-418, latest-419, latest-420 were no longer referenced after removing NX dependencies from 4.22 jobs
Removed unused NODEPOOL vars from autonode job: e2e-aws-autonode only runs ^TestKarpenter which does not use N-minor release images

RCA: Why e2e-*-4-22 jobs were failing (even after revert)

The CI release streams for OCP 4.19, 4.20, and 4.21 have not produced an Accepted build in 4-8 days. The old ci-operator import mechanism (release:latest-NXX) fails hard when any component image within any release payload is missing from the registry — killing the entire job with zero test coverage. Evidence from PR #8673 (post-revert) confirms identical manifest unknown failures via the old mechanism.

The runtime-resolve mechanism is strictly more resilient: a stale image only affects the specific test that uses it, and with the nightly fallback, even stale CI images are recovered automatically.

Full RCA: rca-e2e-4-22-failures.md

Changes

`openshift-hypershift-main.yaml`

Removed n1minor..n4minor release definitions
Removed unused latest-418/419/420 release definitions
Removed OCP_IMAGE_NX dependency overrides from all jobs
Added NODEPOOL_N1_VERSION..NODEPOOL_N4_VERSION env vars to each job (except autonode)

New: `hypershift-resolve-nodepool-releases` step

Queries amd64.ocp.releases.ci.openshift.org API with retry logic (3 attempts × 3 curl retries)
Verifies each resolved image is pullable via oc image info with CI pull credentials
Falls back to nightly stream when CI stream images are unavailable
Writes export OCP_IMAGE_NX="pullspec" to ${SHARED_DIR}/nodepool_release_images
Fails hard with clear error if no stream can provide a pullable image

8 workflow files

Added hypershift-resolve-nodepool-releases step to all e2e workflows (AWS nested/external/metrics/oidc, Azure AKS/AKS-OIDC/self-managed/v2-self-managed)

4 run-e2e scripts + create-guests

Added source ${SHARED_DIR}/nodepool_release_images block (guarded by file existence check)
Changed ${OCP_IMAGE_NX} to nounset-safe ${OCP_IMAGE_NX:-} expansion

Test plan

/pj-rehearse for e2e-aws, e2e-aks, e2e-aws-4-22, e2e-aks-4-22
Verify resolve step returns valid pullspecs in CI environment
Verify nightly fallback activates when CI image is stale
Confirm N1-N4 test flags are correctly passed to the e2e binary

Jira: https://issues.redhat.com/browse/CNTRLPLANE-3509

openshift-ci-robot · 2026-06-10T14:19:33Z

@jparrill: This pull request references CNTRLPLANE-3509 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

Replace ci-operator release imports (n1minor through n4minor) with runtime resolution via the release controller REST API

Fixes the permafail introduced by CNTRLPLANE-3509: Resolve N-minor release images via release controller API #79627 (reverted in Revert "hypershift: resolve nodepool releases at runtime" (#79627) #80349) where unguarded variable expansion under set -o nounset crashed run-e2e scripts before tests could execute

Removes unused latest-418/419/420 release definitions that were left orphaned after removing NX dependencies from 4.22 jobs

Context

PR Status Description

#79627 Merged (Jun 5) Original implementation — caused permafail due to nounset-unsafe ${OCP_IMAGE_NX} expansion

#80349 Merged (Jun 10) Revert of #79627 to restore CI stability

This PR — v2 with fixes for the root cause

What changed vs #79627

Nounset-safe variable expansion: All run-e2e scripts now use ${OCP_IMAGE_NX:-} with -n check instead of bare ${OCP_IMAGE_NX} — prevents crash when resolve step fails or variable is unset

Improved resolve script: Retry logic with exponential backoff, HTTP status code validation, connection/read timeouts, and hard failure with clear error message when resolution fails

Removed orphaned release definitions: latest-418, latest-419, latest-420 were no longer referenced after removing NX dependencies from 4.22 jobs — keeping them caused unnecessary ci-operator import pods

Changes

openshift-hypershift-main.yaml

Removed n1minor..n4minor release definitions

Removed unused latest-418/419/420 release definitions

Removed OCP_IMAGE_NX dependency overrides from all jobs

Added NODEPOOL_N1_VERSION..NODEPOOL_N4_VERSION env vars to each job

New: hypershift-resolve-nodepool-releases step

Queries amd64.ocp.releases.ci.openshift.org API with retry logic (3 attempts × 3 curl retries)

Writes export OCP_IMAGE_NX="pullspec" to ${SHARED_DIR}/nodepool_release_images

Fails hard with clear error if any configured version cannot be resolved

8 workflow files

Added hypershift-resolve-nodepool-releases step to all e2e workflows (AWS nested/external/metrics/oidc, Azure AKS/AKS-OIDC/self-managed/v2-self-managed)

4 run-e2e scripts + create-guests

Added source ${SHARED_DIR}/nodepool_release_images block (guarded by file existence check)

Changed ${OCP_IMAGE_NX} to nounset-safe ${OCP_IMAGE_NX:-} expansion

Test plan

/pj-rehearse to validate affected jobs

Verify resolve step returns valid pullspecs in CI environment

Confirm N1-N4 test flags are correctly passed to the e2e binary

Verify 4.22 pinned jobs (e2e-aks-4-22, e2e-aws-4-22) resolve correctly via API

Jira: https://issues.redhat.com/browse/CNTRLPLANE-3509

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2026-06-10T14:19:41Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

jparrill · 2026-06-10T14:22:22Z

/pj-rehearse ?

coderabbitai · 2026-06-10T14:23:40Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

Walkthrough

Add a hypershift-resolve-nodepool-releases step that queries the release-controller for NODEPOOL_N1..N4, writes exports to ${SHARED_DIR}/nodepool_release_images; switch jobs to accept NODEPOOL_N*_VERSION; scripts source the file and only add --e2e.n*-minor-release-image flags when resolved; workflows insert the new pre-step.

Changes

HyperShift NodePool Release Resolution Mechanism

Layer / File(s)	Summary
Resolve step implementation `ci-operator/step-registry/hypershift/resolve-nodepool-releases/*`	Add `hypershift-resolve-nodepool-releases` step, commands script with multi-stream resolution, pullspec verification, metadata JSON, and OWNERS; script writes `export OCP_IMAGE_N*="..."` to `${SHARED_DIR}/nodepool_release_images`.
CI config: job env and release candidates `ci-operator/config/openshift/hypershift/openshift-hypershift-main.yaml`	Advance `releases.latest` mappings; remove `n1minor..n4minor`; replace OCP_IMAGE_N* dependency wiring with `NODEPOOL_N1_VERSION`–`NODEPOOL_N4_VERSION` job envs across multiple Azure/AWS Hypershift e2e jobs.
Workflow pre-step integrations `ci-operator/step-registry/hypershift/{aws,azure}/*/-workflow.yaml`	Insert `hypershift-resolve-nodepool-releases` into `steps.pre` for multiple AWS and Azure e2e workflows (external, oidc, metrics, nested, self-managed, v2 self-managed, aks/aks-external-oidc).
Scripts and create-guests adaptations `ci-operator/step-registry/hypershift//run-e2e/-commands.sh`, `ci-operator/step-registry/hypershift/azure/create-selfmanaged-guests/*`	Scripts now conditionally `source ${SHARED_DIR}/nodepool_release_images` when present and only construct `--e2e.n-minor-release-image=...` flags if `OCP_IMAGE_N` is set and differs from `OCP_IMAGE_LATEST`. One dependency ref updated from `release:n1minor` to `release:latest`.

Sequence Diagram(s)

sequenceDiagram
  participant Workflow as CI Workflow
  participant ResolveStep as hypershift-resolve-nodepool-releases
  participant ReleaseCtrl as release-controller API
  participant SharedDir as ${SHARED_DIR}
  participant RunE2E as run-e2e / create-guests scripts
  Workflow->>ResolveStep: run pre-step (NODEPOOL_N*_VERSION env)
  ResolveStep->>ReleaseCtrl: request releasestream for NODEPOOL_NX_VERSION
  ReleaseCtrl-->>ResolveStep: return JSON (.pullSpec) or non-200
  ResolveStep->>ResolveStep: retry up to 3 on failure
  ResolveStep->>SharedDir: write export OCP_IMAGE_NX="pullSpec"
  Workflow->>RunE2E: start test step
  RunE2E->>SharedDir: source nodepool_release_images (if exists)
  RunE2E->>RunE2E: add --e2e.n*-minor-release-image flags only if OCP_IMAGE_N* set and != OCP_IMAGE_LATEST

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested labels

lgtm, rehearsals-ack

Suggested reviewers

bryan-cox
sjenning

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error, 1 warning)

Check name	Status	Explanation	Resolution
No-Sensitive-Data-In-Logs	❌ Error	FAIL: run-e2e/resolve scripts enable `set -o xtrace`, which will log internal `RELEASE_CONTROLLER_URL` and pass `--e2e.external-oidc-console-secret=${CONSOLE_CLIENT_SECRET_VALUE}` on the command line.	Disable xtrace (use `set +x` around secret/URL construction and command invocation) or mask/remove secret-valued args so trace output can’t print them.
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (13 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: resolving N-minor release images via the release controller API, with (v2) indicating a revision. It directly reflects the primary objective of the PR.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	PR `#80360` modifies only YAML/SH/OWNERS/JSON files (no .go/.test files shown), so there are no Ginkgo test titles in the changed code to flag.
Test Structure And Quality	✅ Passed	Repo contains 0 Go files with ginkgo/gomega and 11 *_test.go files with no ginkgo patterns; PR appears to touch only CI YAML/scripts, so no Ginkgo tests to review.
Microshift Test Compatibility	✅ Passed	PR changes only CI/operator YAML, shell scripts, and OWNERS/metadata; no new/modified Ginkgo (Go) e2e tests or MicroShift-incompatible API calls to assess.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	PR `#80360` only changes ci-operator CI YAML/SH/OWNERS/JSON (no *_test.go/Ginkgo files), so no new SNO-compatibility test assumptions to flag.
Topology-Aware Scheduling Compatibility	✅ Passed	PR `#80360` only updates hypershift CI step registry/workflows/scripts; scanning all 19 changed files found no scheduling-constraint keywords (affinity/nodeSelector/topologySpreadConstraints/podAntiA...
Ote Binary Stdout Contract	✅ Passed	PR `#80360` changes only CI/YAML and shell scripts (19 files listed); no OTE/openshift-tests Go or binary code was modified, so the OTE stdout JSON contract is not impacted.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	PR `#80360` changes only .yaml/.sh/.json files and contains no *_test.go or Ginkgo test code to flag for IPv4/Internet assumptions.
No-Weak-Crypto	✅ Passed	Scanned the PR’s referenced hypershift CI YAML/shell files for MD5/SHA1/DES/RC4/3DES/Blowfish/ECB and openssl weak-crypto patterns; no matches found, and no custom crypto logic was detected.
Container-Privileges	✅ Passed	Searched the PR’s changed files on GitHub for privileged/hostPID/hostNetwork/hostIPC/SYS_ADMIN/allowPrivilegeEscalation/runAsUser/securityContext; no matches found.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

openshift-merge-bot · 2026-06-10T14:24:36Z

@openshift-ci-robot: your /pj-rehearse request was not processed because the request waited in queue for longer than 5 minutes. Please retry in a few minutes.

openshift-merge-bot · 2026-06-10T14:24:44Z

@openshift-ci[bot]: your /pj-rehearse request was not processed because the request waited in queue for longer than 5 minutes. Please retry in a few minutes.

openshift-merge-bot · 2026-06-10T14:25:51Z

@jparrill: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

openshift-merge-bot · 2026-06-10T14:29:10Z

@jparrill: job(s): , ? either don't exist or were not found to be affected, and cannot be rehearsed

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@ci-operator/config/openshift/hypershift/openshift-hypershift-main.yaml`:
- Around line 529-532: The nodepool resolver always uses the amd64 controller
URL (RELEASE_CONTROLLER_URL) and ignores AWS_MULTI_ARCH, causing amd64-only
pullspecs for multi-arch runs; update the
hypershift-resolve-nodepool-releases-commands.sh logic to check AWS_MULTI_ARCH
(or the equivalent env var) and, when true, set RELEASE_CONTROLLER_URL to the
multi-arch release controller endpoint (or invoke the release lookup with the
multi-arch controller or an arch-appropriate query) before resolving
NODEPOOL_N1_VERSION..NODEPOOL_N4_VERSION so that OCP_IMAGE_N1..N4 are resolved
to multi-arch/appropriate-arch pullspecs consumed by the run-e2e steps. Ensure
the switch references the existing RELEASE_CONTROLLER_URL variable and the
NODEPOOL_N{1-4}_VERSION -> OCP_IMAGE_N{1-4} resolution code paths so no other
behavior changes.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 93129d0d-77bb-43ce-9323-e8fa329f3033

📥 Commits

Reviewing files that changed from the base of the PR and between c60f70c and 3ecbdb1.

📒 Files selected for processing (19)

ci-operator/config/openshift/hypershift/openshift-hypershift-main.yaml
ci-operator/step-registry/hypershift/aws/e2e/external/hypershift-aws-e2e-external-workflow.yaml
ci-operator/step-registry/hypershift/aws/e2e/external/oidc/hypershift-aws-e2e-external-oidc-workflow.yaml
ci-operator/step-registry/hypershift/aws/e2e/metrics/hypershift-aws-e2e-metrics-workflow.yaml
ci-operator/step-registry/hypershift/aws/e2e/nested/hypershift-aws-e2e-nested-workflow.yaml
ci-operator/step-registry/hypershift/aws/run-e2e/external/hypershift-aws-run-e2e-external-commands.sh
ci-operator/step-registry/hypershift/aws/run-e2e/nested/hypershift-aws-run-e2e-nested-commands.sh
ci-operator/step-registry/hypershift/azure/aks/e2e/hypershift-azure-aks-e2e-workflow.yaml
ci-operator/step-registry/hypershift/azure/aks/external-oidc/hypershift-azure-aks-external-oidc-workflow.yaml
ci-operator/step-registry/hypershift/azure/create-selfmanaged-guests/hypershift-azure-create-selfmanaged-guests-commands.sh
ci-operator/step-registry/hypershift/azure/create-selfmanaged-guests/hypershift-azure-create-selfmanaged-guests-ref.yaml
ci-operator/step-registry/hypershift/azure/e2e/self-managed/hypershift-azure-e2e-self-managed-workflow.yaml
ci-operator/step-registry/hypershift/azure/e2e/v2-self-managed/hypershift-azure-e2e-v2-self-managed-workflow.yaml
ci-operator/step-registry/hypershift/azure/run-e2e-self-managed/hypershift-azure-run-e2e-self-managed-commands.sh
ci-operator/step-registry/hypershift/azure/run-e2e/hypershift-azure-run-e2e-commands.sh
ci-operator/step-registry/hypershift/resolve-nodepool-releases/OWNERS
ci-operator/step-registry/hypershift/resolve-nodepool-releases/hypershift-resolve-nodepool-releases-commands.sh
ci-operator/step-registry/hypershift/resolve-nodepool-releases/hypershift-resolve-nodepool-releases-ref.metadata.json
ci-operator/step-registry/hypershift/resolve-nodepool-releases/hypershift-resolve-nodepool-releases-ref.yaml

csrwng · 2026-06-10T14:39:31Z

/lgtm

jparrill · 2026-06-10T14:40:41Z

/pj-rehearse pull-ci-openshift-cloud-network-config-controller-release-4.22-hypershift-e2e-aks

jparrill · 2026-06-10T14:41:18Z

/pj-rehearse pull-ci-openshift-cluster-csi-snapshot-controller-operator-release-4.22-hypershift-aws-e2e-external

jparrill · 2026-06-10T14:41:30Z

/pj-rehearse pull-ci-openshift-aws-karpenter-provider-aws-release-4.22-e2e-hypershift

openshift-merge-bot · 2026-06-10T14:43:29Z

@jparrill: pj-rehearse could not automatically process this event because the request waited in queue for longer than 5 minutes. Use /pj-rehearse to trigger rehearsals manually.

openshift-merge-bot · 2026-06-10T14:45:24Z

@jparrill: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

openshift-merge-bot · 2026-06-10T14:46:21Z

@jparrill: your /pj-rehearse request was not processed because the request waited in queue for longer than 5 minutes. Please retry in a few minutes.

openshift-merge-bot · 2026-06-10T14:46:33Z

@jparrill: your /pj-rehearse request was not processed because the request waited in queue for longer than 5 minutes. Please retry in a few minutes.

openshift-merge-bot · 2026-06-10T14:46:34Z

@openshift-ci[bot]: your /pj-rehearse request was not processed because the request waited in queue for longer than 5 minutes. Please retry in a few minutes.

openshift-merge-bot · 2026-06-10T14:50:27Z

@openshift-merge-bot[bot]: your /pj-rehearse request was not processed because the request waited in queue for longer than 5 minutes. Please retry in a few minutes.

openshift-merge-bot · 2026-06-10T14:51:24Z

@openshift-merge-bot[bot]: your /pj-rehearse request was not processed because the request waited in queue for longer than 5 minutes. Please retry in a few minutes.

openshift-merge-bot · 2026-06-10T14:51:36Z

@openshift-merge-bot[bot]: your /pj-rehearse request was not processed because the request waited in queue for longer than 5 minutes. Please retry in a few minutes.

openshift-merge-bot · 2026-06-10T14:51:36Z

@openshift-merge-bot[bot]: your /pj-rehearse request was not processed because the request waited in queue for longer than 5 minutes. Please retry in a few minutes.

openshift-merge-bot · 2026-06-10T14:56:27Z

@openshift-merge-bot[bot]: your /pj-rehearse request was not processed because the request waited in queue for longer than 5 minutes. Please retry in a few minutes.

openshift-merge-bot · 2026-06-10T14:56:38Z

@openshift-merge-bot[bot]: your /pj-rehearse request was not processed because the request waited in queue for longer than 5 minutes. Please retry in a few minutes.

openshift-merge-bot · 2026-06-12T12:31:29Z

@jparrill: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

openshift-merge-bot · 2026-06-12T12:31:31Z

@jparrill: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

… API Replace static ci-operator release imports (n1minor through n4minor) with runtime resolution via the release controller REST API. This eliminates concurrent import pods that caused race conditions in ci-tools, failing ~40-45% of e2e runs before tests executed. Changes: - Remove n1minor..n4minor and unused latest-418/419/420 release definitions - Replace OCP_IMAGE_NX dependencies with NODEPOOL_NX_VERSION env vars - Add hypershift-resolve-nodepool-releases step to all 8 e2e workflows - Add nounset-safe variable expansion (${OCP_IMAGE_NX:-}) in run-e2e scripts to prevent crashes when variables are unset - Improve resolve script with retry logic, HTTP status checks, and hard failure on resolution errors Fixes the permafail introduced by openshift#79627 where unguarded variable expansion under set -o nounset crashed run-e2e scripts before tests could execute. Jira: CNTRLPLANE-3509 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Juan Manuel Parrilla Madrid <jparrill@redhat.com>

openshift-merge-bot · 2026-06-12T12:56:16Z

[REHEARSALNOTIFIER]
@jparrill: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
pull-ci-openshift-csi-operator-main-hypershift-aws-e2e-external	openshift/csi-operator	presubmit	Registry content changed
pull-ci-openshift-csi-operator-release-5.1-hypershift-aws-e2e-external	openshift/csi-operator	presubmit	Registry content changed
pull-ci-openshift-csi-operator-release-5.0-hypershift-aws-e2e-external	openshift/csi-operator	presubmit	Registry content changed
pull-ci-openshift-csi-operator-release-4.23-hypershift-aws-e2e-external	openshift/csi-operator	presubmit	Registry content changed
pull-ci-openshift-csi-operator-release-4.22-hypershift-aws-e2e-external	openshift/csi-operator	presubmit	Registry content changed
pull-ci-openshift-csi-operator-release-4.21-hypershift-aws-e2e-external	openshift/csi-operator	presubmit	Registry content changed
pull-ci-openshift-csi-operator-release-4.20-hypershift-aws-e2e-external	openshift/csi-operator	presubmit	Registry content changed
pull-ci-openshift-csi-operator-release-4.19-hypershift-aws-e2e-external	openshift/csi-operator	presubmit	Registry content changed
pull-ci-openshift-csi-operator-release-4.18-hypershift-aws-e2e-external	openshift/csi-operator	presubmit	Registry content changed
pull-ci-openshift-csi-operator-release-4.17-hypershift-aws-e2e-external	openshift/csi-operator	presubmit	Registry content changed
pull-ci-openshift-csi-operator-release-4.16-hypershift-aws-e2e-external	openshift/csi-operator	presubmit	Registry content changed
pull-ci-openshift-csi-operator-release-4.15-hypershift-aws-e2e-external	openshift/csi-operator	presubmit	Registry content changed
pull-ci-openshift-csi-operator-main-hypershift-e2e-aks	openshift/csi-operator	presubmit	Registry content changed
pull-ci-openshift-csi-operator-release-5.1-hypershift-e2e-aks	openshift/csi-operator	presubmit	Registry content changed
pull-ci-openshift-csi-operator-release-5.0-hypershift-e2e-aks	openshift/csi-operator	presubmit	Registry content changed
pull-ci-openshift-csi-operator-release-4.23-hypershift-e2e-aks	openshift/csi-operator	presubmit	Registry content changed
pull-ci-openshift-csi-operator-release-4.22-hypershift-e2e-aks	openshift/csi-operator	presubmit	Registry content changed
pull-ci-openshift-csi-operator-release-4.21-hypershift-e2e-aks	openshift/csi-operator	presubmit	Registry content changed
pull-ci-openshift-csi-operator-release-4.20-hypershift-e2e-aks	openshift/csi-operator	presubmit	Registry content changed
pull-ci-openshift-csi-operator-release-4.19-hypershift-e2e-aks	openshift/csi-operator	presubmit	Registry content changed
pull-ci-openshift-cloud-network-config-controller-main-hypershift-e2e-aks	openshift/cloud-network-config-controller	presubmit	Registry content changed
pull-ci-openshift-cloud-network-config-controller-release-5.1-hypershift-e2e-aks	openshift/cloud-network-config-controller	presubmit	Registry content changed
pull-ci-openshift-cloud-network-config-controller-release-5.0-hypershift-e2e-aks	openshift/cloud-network-config-controller	presubmit	Registry content changed
pull-ci-openshift-cloud-network-config-controller-release-4.23-hypershift-e2e-aks	openshift/cloud-network-config-controller	presubmit	Registry content changed
pull-ci-openshift-cloud-network-config-controller-release-4.22-hypershift-e2e-aks	openshift/cloud-network-config-controller	presubmit	Registry content changed

A total of 604 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

jparrill · 2026-06-12T13:11:58Z

/pj-rehearse pull-ci-openshift-hypershift-main-e2e-aws

jparrill · 2026-06-12T13:11:59Z

/pj-rehearse pull-ci-openshift-hypershift-main-e2e-aks

jparrill · 2026-06-12T13:12:00Z

/pj-rehearse pull-ci-openshift-hypershift-main-e2e-aws-4-22

openshift-merge-bot · 2026-06-12T13:12:01Z

@jparrill: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

openshift-merge-bot · 2026-06-12T13:12:02Z

@jparrill: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

jparrill · 2026-06-12T13:12:02Z

/pj-rehearse pull-ci-openshift-hypershift-main-e2e-aks-4-22

openshift-merge-bot · 2026-06-12T13:14:35Z

@jparrill: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

openshift-merge-bot · 2026-06-12T13:15:59Z

@jparrill: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

csrwng · 2026-06-12T13:39:10Z

/lgtm

openshift-ci · 2026-06-12T13:40:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: csrwng, jparrill

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~ci-operator/config/openshift/hypershift/OWNERS~~ [csrwng,jparrill]
~~ci-operator/step-registry/hypershift/OWNERS~~ [csrwng,jparrill]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jparrill · 2026-06-12T16:07:56Z

/pj-rehearse ack

openshift-merge-bot · 2026-06-12T16:07:59Z

@jparrill: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

jparrill · 2026-06-12T16:08:10Z

Errors non related with the PR.

openshift-ci · 2026-06-12T16:27:13Z

@jparrill: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/rehearse/openshift/hypershift/main/e2e-aws-4-22	`8a3b0cb`	link	unknown	`/pj-rehearse pull-ci-openshift-hypershift-main-e2e-aws-4-22`
ci/rehearse/openshift/hypershift/main/e2e-aks	`8a3b0cb`	link	unknown	`/pj-rehearse pull-ci-openshift-hypershift-main-e2e-aks`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

… API (openshift#80360) Replace static ci-operator release imports (n1minor through n4minor) with runtime resolution via the release controller REST API. This eliminates concurrent import pods that caused race conditions in ci-tools, failing ~40-45% of e2e runs before tests executed. Changes: - Remove n1minor..n4minor and unused latest-418/419/420 release definitions - Replace OCP_IMAGE_NX dependencies with NODEPOOL_NX_VERSION env vars - Add hypershift-resolve-nodepool-releases step to all 8 e2e workflows - Add nounset-safe variable expansion (${OCP_IMAGE_NX:-}) in run-e2e scripts to prevent crashes when variables are unset - Improve resolve script with retry logic, HTTP status checks, and hard failure on resolution errors Fixes the permafail introduced by openshift#79627 where unguarded variable expansion under set -o nounset crashed run-e2e scripts before tests could execute. Jira: CNTRLPLANE-3509 Signed-off-by: Juan Manuel Parrilla Madrid <jparrill@redhat.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 10, 2026

openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 10, 2026

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 10, 2026

coderabbitai Bot reviewed Jun 10, 2026

View reviewed changes

Comment thread ci-operator/config/openshift/hypershift/openshift-hypershift-main.yaml Outdated

jparrill force-pushed the CNTRLPLANE-3509-v2 branch from 3ecbdb1 to cb94f24 Compare June 10, 2026 14:38

openshift-ci Bot assigned csrwng Jun 10, 2026

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jun 10, 2026

jparrill force-pushed the CNTRLPLANE-3509-v2 branch from 0b93c2f to 8a3b0cb Compare June 12, 2026 12:52

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jun 12, 2026

openshift-merge-bot Bot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Jun 12, 2026

openshift-merge-bot Bot merged commit 76ba986 into openshift:main Jun 12, 2026
18 of 20 checks passed

Uh oh!

Conversation

jparrill commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Context

What changed vs #79627

RCA: Why e2e-*-4-22 jobs were failing (even after revert)

Changes

openshift-hypershift-main.yaml

New: hypershift-resolve-nodepool-releases step

8 workflow files

4 run-e2e scripts + create-guests

Test plan

Uh oh!

openshift-ci-robot commented Jun 10, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Context

What changed vs #79627

Changes

openshift-hypershift-main.yaml

New: hypershift-resolve-nodepool-releases step

8 workflow files

4 run-e2e scripts + create-guests

Test plan

Uh oh!

openshift-ci Bot commented Jun 10, 2026

Uh oh!

jparrill commented Jun 10, 2026

Uh oh!

coderabbitai Bot commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Suggested labels

Suggested reviewers

Pre-merge checks failed

❌ Failed checks (1 error, 1 warning)

Uh oh!

openshift-merge-bot Bot commented Jun 10, 2026

Uh oh!

openshift-merge-bot Bot commented Jun 10, 2026

Uh oh!

openshift-merge-bot Bot commented Jun 10, 2026

Uh oh!

openshift-merge-bot Bot commented Jun 10, 2026

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

csrwng commented Jun 10, 2026

Uh oh!

jparrill commented Jun 10, 2026

Uh oh!

jparrill commented Jun 10, 2026

Uh oh!

jparrill commented Jun 10, 2026

Uh oh!

openshift-merge-bot Bot commented Jun 10, 2026

Uh oh!

openshift-merge-bot Bot commented Jun 10, 2026

Uh oh!

openshift-merge-bot Bot commented Jun 10, 2026

Uh oh!

openshift-merge-bot Bot commented Jun 10, 2026

Uh oh!

openshift-merge-bot Bot commented Jun 10, 2026

Uh oh!

openshift-merge-bot Bot commented Jun 10, 2026

Uh oh!

openshift-merge-bot Bot commented Jun 10, 2026

Uh oh!

openshift-merge-bot Bot commented Jun 10, 2026

Uh oh!

openshift-merge-bot Bot commented Jun 10, 2026

jparrill commented Jun 10, 2026 •

edited

Loading

`openshift-hypershift-main.yaml`

New: `hypershift-resolve-nodepool-releases` step

openshift-ci-robot commented Jun 10, 2026 •

edited by atlassian Bot

Loading

`openshift-hypershift-main.yaml`

New: `hypershift-resolve-nodepool-releases` step

coderabbitai Bot commented Jun 10, 2026 •

edited

Loading